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Abstract 

Background: Vox settings with limited laboratory capacity, 2013 World Health Organization (WHO) guidelines recommend 
targeted HIV-1 viral load (VL) testing to identify virological failure. We previously developed and validated a clinical 
prediction score (CPS) for targeted VL testing, relying on clinical, adherence and laboratory data. While outperforming the 
WHO failure criteria, it required substantial calculation and review of all previous laboratory tests. In response, we developed 
four simplified, less error-prone and broadly applicable CPS versions that can be done 'on the spot'. 

Methodology/Principal: Findings From May 2010 to June 2011, we validated the original CPS in a non-governmental 
hospital in Phnom Penh, Cambodia applying the CPS to adults on first-line treatment >1 year. Virological failure was 
defined as a single VL >1000 copies/ml. The four CPSs included CPS1 with 'current CD4 count' instead of %-decline-from- 
peak CD4; CPS2 with hemoglobin measurements removed; CPS3 having 'decrease in CD4 count below baseline value' 
removed; CPS4 was purely clinical. Score development relied on the Spiegelhalter/Knill-Jones method. Variables 
independently associated with virological failure with a likelihood ratio >1.5 or <0.67 were retained. CPS performance 
was evaluated based on the area-under-the-ROC-curve (AUROC) and 95% confidence intervals (CI). The CPSs were validated 
in an independent dataset. A total of 1490 individuals (56.6% female, median age: 38 years (interquartile range (IQR 33-44)); 
median baseline CD4 count: 94 cells/ul (IQR 28-205), median time on antiretroviral therapy 3.6 years (IQR 2.1-5.1)), were 
included. Forty-five 45 (3.0%) individuals had virological failure. CPS1 yielded an AUROC of 0.69 (95% CI: 0.62-0.75) in 
validation, CPS2 an AUROC of 0.68 (95% CI: 0.62-0.74), and CPS3, an AUROC of 0.67 (95% CI: 0.61-0.73). The purely clinical 
CPS4 performed poorly (AUROC-0.59; 95% CI: 0.53-0.65). 

Conclusions: Simplified CPSs retained acceptable accuracy as long as current CD4 count testing was included. Ease of field 
application and field accuracy remains to be defined. 
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Introduction 

Scaling-up of antiretroviral treatment (ART) is currently 
ongoing in low and middle income countries (LMIC), arming to 
initiate 15 million individuals on ART by 2015 [1]. One of the key 
challenges for program managers and policy makers in these 
countries is how to monitor these individuals for treatment failure, 
considering the often limited financial resources at hand [2]. 

Routine viral load (VL) testing is now recommended by the 
World Health Organization (WHO) [3] but, with currently 
available technologies, comes at a high cost and is technically 
demanding. Thus, it will still take many years before this will be 
readily available in routine program settings in many LIMC [4,5] . 
Instead, for settings with limited VL testing capacity, the 2013 
WHO guidelines recommend targeted VL testing [3] . 



Targeted VL testing, whereby VL testing is done only in 
individuals meeting failure criteria, aims to avoid unnecessary and 
costly switches to second line treatment for patients with false- 
positive 'screening' tests [2]. Effective implementation of such a 
strategy requires accurate and evidence-based tools to target VL 
testing. Whereas most programs have been applying WHO 
clinical and immunological failure criteria [6], studies have 
consistently demonstrated the low sensitivity and specificity of 
these criteria [2,7]. 

We previously developed a clinical prediction score (CPS) for 
virological failure integrating clinical, adherence and laboratory 
data [8]. At the same time, we constructed an algorithm 
combining the CPS with targeted VL testing (in patients with a 
CPS ^2). The score performed substantially better than the WHO 
failure criteria, and performed well in internal validation 
(Cambodia) and external validation (Uganda) [9,10]. Experience 
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from Lesotho provided additional support for its use in patients 
who were identified based on WHO immunological and clinical 
criteria as treatment failure [11]. A few other CPSs have been 
developed for the same purpose, but they have either not been 
validated, or performed poorly during validation [9,12,13]. 

Two limitations of the original CPS were identified during the 
validation study in Cambodia [10]. Firstiy, frequent errors were 
made when physicians applied the score, which affected the CPS 
performance. Errors in scoring of the individual items were most 
commonly seen for items like the percentage decrease from peak 
CD4 cell counts or the decline in hemoglobin values since these 
items require calculation, and availability and review of all 
previous laboratory results. Secondly, the reliance on regular 
laboratory monitoring of CD4 count and hemoglobin values limit 
implementation of the original CPS in settings where such tests are 
not routinely performed. In response to these limitations, we 
derived and validated several simplified versions of the original 
clinical score, aiming to make the tool less error-prone, easier to 
apply and more broadly applicable. 

Methods 

Study Setting 

Sihanouk Hospital Center of HOPE (SHCH) is a non- 
governmental hospital in Phnom Penh Cambodia. Since 2003, 
the hospital has provided ART at no cost as part of the national 
program. Patients were initiated and treated according to WHO 
recommendations [6,14]. First line treatment consisted of a 
generic combination of stavudine, lamuvidine and nevirapine. 
Zidovudine and efavirenz was used in case of contraindications. 
The revised 2010 guidelines were implemented in May 2010. 
Patients were seen at regular intervals for clinical and laboratory 
monitoring and adherence assessment. All care was provided by 
physicians. More information on the program has been published 
before [15-17]. 

Validation Study of the Original CPS 

Details of the original validation study have been previously 
reported [10]. In brief, we conducted a cross-sectional study within 
an established ART cohort in Cambodia between May 10, 2010 
and June 3, 2011, including all consenting adults on a non- 
nucleoside containing (standard) first-line regimen for a minimum 
of one year (n = 1490). The original score - derived in this hospital 
- contained eight items: 1) CD4 decline from peak >25%, 2) CD4 
decline from peak >50%, 3) hemoglobin drop^l g/dL over the 
last six months, 4) current CD4 below baseline, 5) CD4 count < 
100 cells/|J.L at 12 month of ART, 6) papular pruritic eruption 
(PPE), 7) treatment adherence <95% (measured using a visual 
analogue score); 8) ART-experience [8]. Summing the predictor 
scores of the individual's risk factors yielded the total predictor 
score for each patient. Virological failure was defined as a VL 
above 1000 copies/ml at the cross-sectional survey. The overall 
test performance was assessed by calculating the area under the 
receiver-operating characteristic curve (AUROC). 

Development and Validation of Simplified CPS 

The following simplified CPSs were evaluated with a progres- 
sive increase in simplification by removing: 1) the 'decline from 
peak CD4 count values' and only using 'current' CD4 count 
(CPS1) - ie the value of the CD4 count at the time of assessment 
for virological failure; 2) hemoglobin as item in the CPS (CPS2); 3) 
'decrease in CD4 count below baseline value' as item in the CPS 
(CPS3); and 4) all laboratory items from the CPS (CPS4). 



The same methodology for score development was used as for 
the original score derivation [8] . In brief, we used the Spiegelhalter 
and Knill-Jones method adapted by Berkley et al [18-20]. 
Continuous variables were dichotomized as guided by ROC 
curves. Variables independently associated with virological failure 
with a likelihood ratio (LHR) £1.5 or £0.67 were retained. The 
score of each predictor was calculated as the natural logarithm of 
the adjusted LHR, rounded to the nearest integer. Summing the 
predictor scores of the individual's risk factors yielded the total 
predictor score for each patient. For simplification, the score was 
recoded by setting the reference level for each predictor to zero 
[20]. The performance of the different CPSs was based on the 
AUROC and 95% confidence intervals (CIs). A CPS was 
considered clinically useful if, during validation, the lower limit 
of the 95% CI was 0.6 or above. In addition, we calculated the 
sensitivity, specificity, positive and negative predictive values, the 
positive and negative likelihood ratio's and proportion of 
individuals that would require VL testing at the different cut-off 
values of each CPS. The simplified scores were developed using 
the same dataset of the study mentioned above, conducted in 
2010-11. The different CPSs were validated in an independent 
dataset from the same hospital (2005-2007), as reported before 
[8]. In this study, patients were enrolled following the same 
inclusion criteria as the 2010-2011 study mentioned above 
(Table 1). Confidence intervals for the validated AUROCs were 
calculated using robust standard errors. 

Ethical Considerations 

The VL validation study was approved by the National Ethics 
Committee for Health Research, Phnom Penh, Cambodia; the 
institutional review board (IRB) of the Institute of Tropical 
Medicine, Antwerp, Belgium; and the Ethics Committee of the 
University Hospital of Antwerp, Belgium. All study participants 
provided written consent. 

Results 

A total of 1490 adults on first-line ART for at least one year 
were included. As previously described, median time on ART at 
the time of evaluation of virological treatment response was 3.6 
(IQR 2.5-5.1) years. The median age at study inclusion was 38 
(IQR 33-44) years and the median CD4 count was 379 (IQR 
265-507) cells/uL. Approximately half (56.7%) were female and 
>90% were on a zidovudine or stavudine-based regimen at the 
time of evaluation. A total of 45 patients (3.0%) were detected with 
virological failure. 

In a first step (CPS1), we removed CD4 decline from peak 
value, since it requires substantial calculation and complete on 
treatment CD4 count data, but retained 'current CD4 count'. The 
AUROC of CPS1 was 0.78 (95% CI 0.70-0.85) in derivation and 
0.69 (95% CI 0.62-0.75) in validation (Table 2). With the WHO 
2013 guidelines arguing against routine hemoglobin monitoring - 
even for patients using zidovudine - hemoglobin measurement was 
additionally removed in CPS2. The validated AUROC decreased 
to 0.68 (95% CI 0.62-0.74). To obtain a tool devoid of any 
requirement of previous laboratory results (hence only relying on 
"current" lab tests), comparison with baseline CD4 count was 
removed in CPS3. Only a slight decrease in AUROC was 
observed. This provided a system essentially relying on basic 
clinical information collected during routine patient assessment: 
clinical evaluation (WHO T-stage, PPE, treatment information 
(ART-experience and ART adherence) and CD4 response at the 
time of assessment. No laboratory monitoring was included in 
CPS4. However, this tool performed poorly (AUROC 0.59 in 
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Table 1. Datasets used for the development and validation of the simplified prediction score to identify virological failure. 







Derivation dataset" 


Validation dataset a b 




2010-2011 


2005-2007 




1490 individuals 


1803 episodes 


Age (years) at baseline, median (IQR) 


35 (30-41) 


34 (18-68) 


Male sex, n (%) 


646 (43.4) 


925 (51.3) 


Years on ART at viral load assessment, median (IQR) 


3.6 (2.1-5.1) 


1.5 (1.0-2.0) 


Virological failure - n (%) 


45 (3.0) 


77 (4.8) 


Initial ART regimen 


D4T-3TC-NVP 


1 056 (70.9) 


1369 (75.9) 


D4T-3TC-EFV 


343 (23.0) 


261 (14.5) 


AZT-3TC-NVP 


62 (4.2) 


106 (5.9) 


AZT-3TC-EFV 


24 (1.6) 


63 (3.5) 


Other 


5 (0.4) 


4 (0.2) 


CD4 count, cells/|xL, median (IQR) 


Baseline (ART initiation) 


94 (28-205) 


54 (15-147) 


At viral load assessment 


379 (265-507) 


281 (187-398) 



inclusion criteria: consecutive enrolment of adult patients (>18 years old) in follow-up at the Sihanouk Hospital Center of Hope on first-line ART for at least 12 months; 
viral load was systematically performed for study purpose; virological failure was defined as a single measurement >1000 copies/ml; 
b more than one episodes per individual. 

ART: antiretroviral treatment; AZT: zidovudine; D4T: stavudine; 3TC: lamivudine; NVP: nevirapine; EFV: efavirenz; IQR: interquartile range. 
doi:1 0.1 371 /journal.pone.0087879.t001 



validation). The sensitivity, specificity, positive predictive value Discussion 
and percentage of individuals requiring a VL at various score cut- 
off values of the different CPS is given in Table 3, applying the We developed four simplified CPSs, of which three performed 
CPS in an independent dataset relatively well and have a number of advantages. Compared to the 

original CPS, no regular laboratory monitoring while on ART is 



Table 2. Content and diagnostic performance of the simplified clinical prediction scores to identify virological failure 3 . 







CPS 1 


CPS 2 


CPS 3 


CPS 4 




No CD4 peak values 


No hemoglobin 


No CD4 baseline 


No lab investigations 


AUROC derivation 


0.78 


0.75 


0.75 


0.63 


(95% CI); (n = 1490) 


(0.70-0.85) 


(0.68-0.83) 


(0.67-0.82) 


(0.56-0.70) 


AUROC validation 


0.69 


0.68 


0.67 


0.59 


(95% CI); (n = 1803) 


(0.62-0.75) 


(0.62-0.74) 


(0.61-0.73) 


(0.53-0.65) 


Items 


CD4 decline from peak >25% 


Not included 


Not included 


Not included 


Not included 


CD4 decline from peak >50% 


Not included 


Not included 


Not included 


Not included 


Hemoglobin drop>:1 g/dL b 


+1 


Not included 


Not included 


Not included 


Current CD4 below baseline 


+1 


+1 


Not included 


Not included 


Current CD4<250 cells/|iL 


+2 


+2 


+2 


Not included 


WHO T-stage 'A b 


+1 


+1 


+1 


+1 


pp E b 


+2 


+2 


+2 


+2 


Adherence (VAS) <95% c 


+1 


+1 


+2 


+2 


ART-experience 


+2 


+1 


+1 


+1 



ART: antiretroviral treatment; AUROC: area under the receiver-operating characteristic curve; CPS: clinical prediction score; PPE: papular pruritic eruption; VAS: visual 
analogue scale. 

^differences between CPSs in the score of some predictors relate to the fact that the weight of each remaining predictor was recalculated after removing of specific 
predictors [eg after removing hemoglobin in CPS2 vs CPS1). 
b over a period of six months prior to viral load measurement. 
c over a period of one month prior to viral load measurement. 
doi:1 0.1 371 /journal.pone.0087879.t002 
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Table 3. Diagnostic performance at different cut-offs of the 
clinical prediction scores to identify virological failure using an 
independent dataset. 







Score 


Sensitivity 


Specificity 


PPV 


% VL 
done 


LHR(+) 


LHR(-) 


CPS 7 


21 


80.5 


36.5 


6.0 


64.3 


1.27 


0.54 


sa 


73.6 


50.3 


7.0 


50.8 


1.48 


0.52 


23 


49.4 


80.4 


11.3 


21.1 


2.52 


0.63 


24 


34.5 


92.1 


18.1 


9.2 


4.35 


0.71 


CPS 2 


21 


79.1 


41.0 


6.4 


60.0 


1.34 


0.50 


22 


72.4 


54.8 


7.5 


46.5 


1.60 


0.50 


S3 


41.4 


84.7 


12.1 


16.5 


2.71 


0.69 


24 


24.1 


95.9 


23.1 


5.0 


5.92 


0.79 


CPS 3 


21 


78.2 


41.3 


6.3 


60.0 


1.33 


0.53 


22 


72.4 


54.8 


7.5 


46.5 


1.6 


0.50 


23 


36.8 


85.7 


11.6 


15.4 


2.58 


0.74 


24 


21.8 


96.3 


22.9 


4.6 


5.86 


0.81 


CPS 4 


21 


43.7 


70.0 


6.9 


30.6 


1.46 


0.80 


22 


25.3 


91.5 


13.2 


9.3 


2.99 


0.82 


23 


9.2 


97.8 


17.8 


2.5 


4.26 


0.93 


24 


1.1 


99.9 


33.3 


0.2 


9.86 


0.99 





CPS: clinical prediction score; PPV: positive predictive value; % VL done: 
percentage of patients that would undergo targeted VL testing; LHR: likelihood 
ratio. 

doi:1 0.1 371 /journal.pone.0087879.t003 



required. Moreover, with a choice of CPS, programs can decide 
which is most appropriate and feasible to apply in their program 
settings. Performing hemoglobin measurement as part of an ART 
monitoring strategy can be cumbersome in many settings, comes 
with an additional cost and is not required in the latest WHO 
guidelines. Moreover, the added value in terms of diagnostic 
performance was limited, especially at lower cut-off values. CPS2 
and CPS3 seem most useful and practical. CPS2 could be used 
when a baseline CD4 cell count is available, while CPS3 would 
apply when patients have initiated ART without CD4 guidance. 
With only a limited amount of key clinical information and 
minimal calculation required, these CPSs are also substantially 
simpler and possibly more adapted for use by non-physicians. 

Some resource-constrained countries predominantly use clinical 
monitoring to detect treatment failure and CD4 cell counts are not 
always regularly done while on ART, especially in remote areas. 
However, we found that, in line with previous studies, CPS4 - 
relying on clinical information only - performed poorly [2]. 
Adding a CD4 cell count test 'on the spot' would substantially 
improve the performance of the algorithm without requiring 
routine CD4 monitoring. Current CD4 count was also found to be 
a strong predictor in a Ugandan scoring system [9] . Of interest, 
the same CD4 count cut-off (250 cells/uL) was identified in a 
recent study using data from the DART trial [21]. Pending the 
development of an affordable and widely available point of care 
VL test, use of these CPSs combined with targeted VL testing 
should be considered. If CD4 count testing can be done 'on site' or 
with a point of care test [22], application of the CPS could be done 



within one day. If not, patients might need two visits to apply the 
CPS. However, this creates an additional burden for both patients 
and health care staff, and increases chances for delays and missed 
appointments. 

Further study is required to determine at what time points 
during ART these scores should be applied, and in which different 
health care settings and patient populations. Moreover, these 
algorithms were designed for patients on ART for at least one 
year. One approach could consist of a routine VL early after ART 
initiation (eg six months after ART initiation) followed by 
application of the score at predefined intervals, combined with 
targeted VL testing. Whereas applying the score every six months 
would equate to routine CD4 count monitoring, the current CPS 
would still have the advantage of not requiring interpretation of a 
trend of laboratory tests. While these simplifications would likely 
render them easier to apply and less error-prone compared to the 
original CPS, this is subject to further study. 

A number of issues remain to be assessed before implementing 
these CPSs in routine care. Validation in other study populations 
and different settings should be conducted. Additional studies on 
the clinical utility and impact are warranted [23,24]. The study 
population in this analysis consisted of patients on ART for several 
years. Possibly, the optimal 'current CD4 cell count' cut-off might 
differ in patient populations with short or very long treatment 
duration. Moreover, most patients in our cohort started ART with 
relatively advanced HIV disease. It remains to be assessed whether 
the performance of the CPSs would be different in patients starting 
early ART, as would occur within a test-and-treat strategy. 

Additional limitations include the fact that treatment failure was 
based on a single VL measurement, in line with the previous 
derivation and validation study. Since a substantial proportion 
might have an undetectable VL on repeat measurement, only a 
proportion of those with virological failure in our study would be 
true treatment failures. On the other hand, the main purpose of 
targeted VL testing is early detection of active viral replication, 
since these patients need enhanced adherence counseling followed 
by a repeat VL. We did not evaluate whether individual 
opportunistic infections should be included as predictors in the 
CPS, but used 'pooled' WHO clinical staging groups. While this 
might be a more 'crude' approach, it has the advantage of being 
simpler. Finally, the failure rate in our study was clearly lower than 
what has been observed in most other programs. Consequently, 
positive predictive values will be higher and negative predictive 
values lower in settings with higher failure rates. 

We acknowledge that the present algorithms are unlikely to be 
universally applicable and generalizable across a wide range of 
geographical regions, populations and health care settings. 
Derivation and validation of algorithms for local use (eg at the 
country or regional level) might be a reasonable alternative, and 
might also be valuable for a wider range of diagnostic tests or 
health care interventions. Targeted evaluation using evidence- 
based tools might be a rational way to optimize the use of often 
scarce resources available for health delivery in LMIC [2]. At the 
same time, such an approach also appreciates that patient 
populations are not homogeneous, but consist of different 'risk 
groups' with different needs and benefits of testing and other 
interventions. Especially as long as cheap point of care tests are 
lacking, such approaches merit further study. 

In conclusion, we have developed three simplified CPSs for 
targeted VL testing that retained an acceptable diagnostic 
performance as long as 'spot' CD4 testing was included. They 
have the advantages of not requiring regular CD4 cell count or 
hemoglobin testing, or complete and reliable data collection while 
on ART, or the need to interpret trends in laboratory tests. They 
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should, therefore, be less prone to error and more readily adopted 
in field conditions. Their use would enable more accurate targeted 
VL testing in contexts where it is too expensive to be done 
routinely. We suggest that they be tested and tailored to the other 
health care settings. 
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